2.5.1 Training data.

To create the training dataset needed to develop a classifier via machine learning, a random sample of records returned from the query above was selected for manual review. The training set was created in two phases, with the first containing 638 records and second containing an additional 996 records for a total of 1634 records in the training dataset. Article titles and abstracts were independently reviewed by two curators in each phase and classified as either describing a biodata resource or not describing a biodata resource based on the developed definitions (see above). We kept the entries where both curators agreed on the article classification label (either positive or negative, n = 1587) and used this as a training dataset for the article classification task. For articles manually classified as describing a biodata resource, mentions of biodata resources in the title and abstract were identified, including “common names” (e.g., PDB) and “full names” (e.g., Protein Data Bank). This curated set of mentions was used for the NER task. Both training datasets were split into 70% training, 15% validation, 15% test (hold-out) for article classification (Table 1) and NER tasks (Table 2). Following common practice, the training sets were used to fine-tune the models on their respective tasks, the validation sets were used to compare the fine-tuned models for selection, and the test sets were used to evaluate how the models perform on unseen data [23].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Training dataset splits for the article classification task.

https://doi.org/10.1371/journal.pone.0294812.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Training dataset splits for the Named Entity Recognition (NER) task.

https://doi.org/10.1371/journal.pone.0294812.t002

2.5.2. Models.

Two machine learning models were fine-tuned to automate the process of 1) classifying research articles and 2) extracting mentions of biodata resources from those predicted to describe a biodata resource. Given a paper’s title and abstract, the article classification model classifies the paper as being about a biodata resource or not. If an article receives a positive score, it is then passed through the NER model, which extracts the common name and full name of the biodata resource from the text, if they are present. A confidence score, computed as the average probability among the tokens constituting the mention, is also output (S2 Fig). BERT performs well on a variety of NLP tasks and several BERT derivatives have been pre-trained on biomedical corpora, making them excellent candidates for this project. For both the article classification and NER task, BERT itself and 14 other BERT model variations available on Hugging Face Hub (https://huggingface.co/) were fine-tuned and evaluated to select the highest performing model (Table 3 and citations therein).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Pre-trained models that were fine-tuned for the article classification and NER tasks.

https://doi.org/10.1371/journal.pone.0294812.t003

2.5.3. Article classification task.

The pre-trained models were fine-tuned on the article classification task. The model with the highest performance on the validation set was selected and used to generate the inventory. In order to consider both title and abstract for classification the title and abstract were concatenated with a space character between fields to create a contiguous string. XML tags were removed using regular expressions, while adding white space after punctuation if not present after tag removal. The resulting input string was tokenized using a pre-trained tokenizer associated with the specific pre-trained model to be used for classification. Tokenized input was then passed through the pretrained model module to obtain context embeddings, which were subsequently fed into a linear classification layer that performs binary classification. This process classifies an article, based on title and abstract, as describing a biodata resource or not (Fig 3A).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Machine learning model architectures for article classification and named entity recognition.

Model architecture for (A) sequence classification used to classify articles based on title and abstract and (B) token classification used to perform NER to obtain the resource names. Each architecture shows the possible classification labels resulting from model prediction. Additional details for token classification are shown in S2 Fig.

https://doi.org/10.1371/journal.pone.0294812.g003

Classification performance was evaluated on the validation set using precision (Eq 1), recall (Eq 2), and F1 score (Eq 3). For calculating performance metrics, an article that describes a biodata resource that is correctly classified is a true positive (TP), and if incorrectly classified is a false negative (FN). An article that does not describe a biodata resource that is correctly classified is a true negative (TN), and if incorrectly classified is a false positive (FP). Precision is the proportion of those articles predicted to describe a biodata resource that are indeed describing a biodata resource. Recall is the proportion of articles that describe a biodata resource that are correctly classified. F1 score is the harmonic mean of precision and recall.

(1)

(2)

(3)

All models were trained for a maximum of 10 epochs following general convention as a starting point. For each model the model checkpoint with the highest precision (regardless of epoch number) was saved. After evaluating the inventory in a mid-project evaluation (see section 2.6 below), precision was chosen over F1 and recall for the classification task to reduce the number of false positives. From this evaluation, we found that preferencing precision would improve overall quality of the inventory such that we would be able to finalize the inventory without manually validating every resource included. The precision scores of all the fine-tuned models on the validation set were compared to select the most performant model, which was then used for classification of unlabeled data.

2.5.4. Named entity recognition (NER) task.

The same pre-trained models evaluated for the article classification task were fine-tuned for the NER task (Table 3). To accomplish this, the linear sequence classification layer was replaced by a token classification layer (Fig 3B and S2 Fig). Input (training and validation) data was tagged using the following BIO scheme [33]:

• B-COM: token is the start of a sequence corresponding to a common name
• I-COM: token is a non-start part of a sequence corresponding to a common name
• B-FUL: token is the start of a sequence corresponding to a full name
• I-FUL: token is a non-start part of a sequence corresponding to a full name
• O: outside

NER performance was evaluated using partial-match entity level metrics (precision, recall, F1 score) on the validation set. As with the classification task, models were trained for a maximum of 10 epochs. For each model architecture the trained model checkpoint with the highest F1 score was saved. The model with the highest F1 score on the validation set was then used for downstream tasks.

2.5.5. Model implementation details.

The Huggingface framework was utilized to load the BERT model architectures [34]. The Huggingface’s AutoModelForSequenceClassification module was used to fine-tune the BERT models on the article classification task, and the AutoModelForTokenClassification module to fine-tune the models on the NER task. The hyper-parameters used during fine-tuning of the models can be found in S4 Table. We used the Adam optimizer for training and the seqeval module for computing partial match entity-level metrics. Models were trained for a maximum of 10 epochs. All machine learning code was implemented in Python v3.8.12 and used the following third-party packages: Datasets v1.18.3, Natural Language Toolkit (NLTK) v3.6.1, NumPy v1.19.2, Pandas v1.2.4, pytest v6.2.4, scikit-learn v0.24.1, PyTorch v1.9.0, tqdm v4.63.0, and transformers v4.16.2 [34–42].

2.7.1. Best name determination.

As detailed above, for each article the NER model attempts to predict both full and common names and may output multiple predictions, each of which is associated with a probability score, where the higher the score the greater the confidence in the predicted name. To determine the highest quality names, the probability scores for named entities of each type were compared to determine the “best common name” and “best full name”. These probability scores of these two named entities were compared to choose the best overall name “best name” (that with the highest probability score).

2.7.2. Automated deduplication.

Many resources publish repeatedly, for instance, to provide updates about the resource. Consequently, the raw inventory contained duplicate records from several articles describing the same resource. A first step toward deduplicating the inventory was performed by identifying records that had the same predicted best name and same extracted URL (ignoring differences due to trailing slashes or “http:” vs “https:”). These duplicate resources were merged with the PMIDs of each original article and retained along with the title-abstract text and publication date of the most recently published article.

2.7.3. Annotation for selective manual evaluation.

Up to this point all steps were automated. However, based on the results of the mid-project evaluation described above, we realized that it would be advantageous to conduct a selective manual evaluation of some predicted resources to improve the overall quality of the inventory. In preparation for this step, a script within the pipeline added a new column with the variable “low_prob” for any resource whose best name probability < 0.978, the value determined in the mid-project evaluation as the average of correctly predicted names in the 10% random sample. This served as a flag to aid the curator conducting the manual evaluation. Additionally, while the automated deduplication was able to merge any articles with exact names and exact URLs matches, we were aware of suspected duplicates (e.g. variable names such ’’FANTOM’’ and ’’FANTOM5’’ sharing the same extracted URL while variable URLs such as http://appris-tools.org and http://appris.bioinfo.cnio.es share the same predicted name ’’APPRIS’’). Deduplication on either name or URL (as opposed to both) would have led to erroneous mergers. For example, ’’Seed’’ and ’’SEED’’ are two different resources, while there are at least three distinct ’’PED’’ resources and two distinct resources for ’’SMART.’’ Instead, to account for potential duplicates, columns were also generated for matching best names (flagged with “duplicate_names”) or matching extracted URLs (flagged with “duplicate_urls”) for evaluation by the curator. While more complex, automated procedures may be warranted in the future as the inventory grows over time, these cases were relatively few and we judged this strategy to be sufficient for the time being.

2.7.4. Selective manual evaluation procedure.

With steps implemented above, a preliminary inventory with records flagged for manual review was generated as a CSV file. A curator then reviewed each flag to determine if low probability records should be removed from the inventory and if potential duplicate records should be merged within the inventory. This review was done in Microsoft Excel, with data validation applied to a set of predetermined outcomes (e.g. ’’remove’’, ’’merge’’, etc.). Importantly, no corrections to the predicted names or URLs were made; thus all values within the inventory are the output of the machine learning pipeline, which reduces the confusion that could result if the inventory were a mixture of ML-generated and human-generated names/URLs (especially in future updates). Review guidelines were developed to help standardize handling of edge cases (see S7 Table). The resulting manually reviewed file was added into the directory for subsequent processing. The pipeline first ensures all flagged records contain only valid review values and then removes or merges the appropriate records before moving on to metadata augmentation.

2.7.5. Metadata augmentation.

Once we had the final prediction script and the results of the manual review processed, the Europe PMC API was once again queried using PMIDs to retrieve author affiliations, author names, grant IDs, grant agency name, and citation counts (i.e. via metadata elements ’affiliation’, ’fullName’, ’grantID’, ’agency’, ’citedByCount’, respectively) for each article associated with the biodata resources.

2.7.6. URL processing.

Biodata resources may be impermanent for reasons that include loss of funding, loss of key personnel, and technological change leading to deprecation, and we were interested in establishing if the URL provided in the abstract was still viable. Accordingly, the extracted URLs were checked for viability through standard HTTP status calls using the Python requests and urllib3 (v1.26.8) [43] libraries, and the returns (e.g. 200 OK, 404 Not Found, etc.) were recorded in the inventory on 11 November 2022. Extracted URLs were tested three times, with each attempt allowing 5 seconds for a response. The second attempt is submitted immediately after the first, while the third attempt is submitted after a one second delay.

Web archives offer a chance to locate snapshots of previously available websites [44]. To mitigate current and anticipate future availability issues, we used the Internet Archive’s Wayback Machine API to check URLs for the presence of archived sites. While the Wayback crawler is often unable to access the data itself, these snapshots provide views of HTML pages such as the home page, search interface, etc. which provide important context in the absence of the live site. For the biodata resources in this inventory the most recent Wayback Machine URL was recorded for successful returns; for live URLs not represented in the Wayback Machine the URLs were submitted for archiving and the associated Wayback URL recorded. While the Wayback Machine is able to crawl and archive the majority of sites represented here, sites behind firewalls or those that prohibit crawlers cannot be archived.

2.7.7. Resource geolocation.

We use two methods to identify the location of the biodata resource. The first is the location as determined from the IP address, which suggests a physical location for the infrastructure. For those URLs that return a status less than 400, the IP address is obtained from the host name. In an attempt to geolocate the IP addresses, ipinfo [45] and ip-api [46] are queried to request the IP address country and coordinates. Two APIs are used since neither is complete, and querying multiple APIs increases the chance of successful geolocation. When a location is successfully obtained from either API, the country and coordinates are recorded.

Because of the global nature of the life sciences research enterprise, physical location alone may not reflect collaboratively developed resources. Therefore, we also extracted country names following ISO 3166 from the author affiliations available from the Europe PMC metadata. When geolocating IP addresses both coordinates and country name were returned. ISO-3166-1 Alpha-3 codes and country names of all countries were searched against both the IP address geolocations and author affiliations [47] and recorded in the inventory.

2.9.1. Analysis of funders.

To evaluate the funding agency names found in article metadata, agency names which appeared three or more times were extracted and the countries of origin for the funding body were manually identified through Google searches. Recorded country names were verified by a second curator and then standardized to a three-letter coding following ISO 3166–1 alpha-3. For funders international in nature, which are common for funding provided through the European Union, a unique 3-letter code of “INT” was used in the absence of an ISO standard.

2.9.2. Comparison with existing registries.

To get a sense of how well this machine-learning enabled method of identifying biodata resources may complement other methods, we compared the biodata resources in this inventory with life science data resources collected elsewhere. There are several collections, such as the catalogue maintained by the journal Nucleic Acids Research and Database Commons; however, neither offer bulk download or programmatic access to their data. As such, we focused on two registries, re3data.org [13] and FAIRsharing [14]. Both carefully curate their registered resources and provide robust API access. These registries, which were created for different purposes and via different teams, do not themselves contain all known data resources but inclusion criteria are similar to the inventory (see below). Therefore, we anticipated some overlap but not a complete union with either.

Library and information science professionals in Germany established re3data.org in 2013, specifically as a registry of research data repositories, and the resource has maintained a domain agnostic collection policy. The criteria for inclusion in re3data requires that a resource 1) be run by a legal entity, 2) has access conditions and terms of use, and 3) focuses on research data. We accessed all resource records from re3data’s API and then filtered to align with the inclusion criteria for the inventory. To do this, we subsetted to only resources categorized as “life sciences” which had URLs and removed the generalist categories (specifically “institutional” or “other”) to focus on resources specifically dedicated to the life sciences. FAIRsharing emerged from its more focused predecessor, BioSharing, itself originating from the Minimum Information about a Biomedical or Biological Investigation Portal. Along with data resources, FAIRsharing also includes standards and policies and has expanded beyond the life sciences. FAIRsharing explicitly excludes individual datasets, consistent with the inclusion criteria of the inventory, and requires that resources are 1) organised collections of data, 2) findable via an active website that allows users to browse and/or search, and 3) accessible regardless of specific license type. We retrieved only resource records for the life sciences category from the API and did not filter the results further.

To prepare the resources from re3data, FAIRSharing, and the inventory for comparison, white space was trimmed from the resource name, and resource URLs were cleaned by removing the scheme and any trailing slashes and then converting to lowercase. As noted above, resource names and URLs may vary slightly but significantly, so we compared by common name, full name when available, and URL separately, merged the results, and then deduplicated where both the name and URL matched to obtain the final number of resources found in common.

3.3.1. Mid-project manual evaluation.

In the mid-project evaluation of a 10% sample, 468 articles were manually reviewed. Of these, we found that 439/468 (0.938) articles were classified correctly. In Fig 5A, we show an example of a correctly classified article which was relatively straightforward while in Fig 5B, the text describes an entity that appears, on human reading, to be a tool that does not provide access to data [68, 69]. Another scenario that proved challenging for the models was when the title-abstract described a research project for which data was deposited into a resource (such as Flybase [70]) but the article did not describe the resource itself. However, overall these errors were limited, and given the challenge of classification even for human curators, we judged that the machine learning based methodology was indeed viable for classification.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5.

Examples of (A) a correctly classified article and (B) an incorrectly classified article.

https://doi.org/10.1371/journal.pone.0294812.g005

In addition to evaluating the correctness of classifications, in the mid-project evaluation we looked at the results of the NER extraction for all 468 articles of the 10% sample to determine if extracted common and full names were correct, partially correct, or incorrect. A common name was predicted for 457 of the 468 (97.6%) in the sample, and in 425 of these predictions (93.0%), the common name with the highest probability was, in fact, the correct common name. Notably, even when not correct, only 10 (2.2%) were entirely incorrect (e.g., predicted as “the” instead of “WikiPathways Database”) while 22 (4.8%) were partially correct (e.g., predicted as “Open” instead of “Open TG-GATEs”). In reality, for several that were entirely incorrect, the name was so poorly articulated in the abstract that it was difficult for a human to determine a valid name. Full names were predicted less frequently on the whole (157/468, 33.5%) and, reflective of greater complexity (see Fig 6 for an example), were less likely to be judged correct with 97 being correct (61.8%). However, only 7 (4.5%) were entirely incorrect and 53 were partially correct (33.8%). From these results, it was clear to us that, as was true with predicted classification, the method was viable for predicting resource names, with common names being the most likely identified.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Example of correct extraction of common and partial extraction of full name.

Example of a detailed abstract where the common name of the resource, “ESTHER” (bold/blue), was only mentioned once but correctly predicted. The predicted full name, “Hydrolase” (bold/orange), was partially correct while the entire full name (peach) is much longer. The uncertainty is accurately reflected in the scores where the correctly predicted common name is associated with a probability of 0.9933 while the partially correct full name was lower at 0.7105.

https://doi.org/10.1371/journal.pone.0294812.g006

While it was clear from the results of the mid-project evaluation that the machine learning methods would be a useful technique to employ, we also recognized that the quality of inventory, as a collection of discrete resources, is important to the stakeholders. We explored putting greater weight on precision than F1 for both the classification and NER tasks, but we found that the subsequent hit to recall in the NER task resulted in the loss of too many viable predictions. To achieve higher reliability for the inventory then we choose a selectively mediated approach given the relatively small size of the inventory and manageable curation load. Based on the mid-project evaluation results we used the average probability of correctly predicted names, 0.978, to determine a threshold at which resources whose highest scoring predicted name (“best name,” regardless if common or full) had a probability below this threshold would be flagged for review by a curator in our finalized process. In the 468 articles from the 10% random sample, records with probabilities ≥ 0.978 (i.e. those proposed to not undergo human-mediated review in the future) contained 13/468 (2.8%) incorrect classifications and 5/468 (1.0%) incorrect best names. Therefore, we concluded that a selective manual evaluation at this threshold was likely to catch the majority of both classification errors and name errors and improve the quality of the final inventory.

In the mid-project evaluation, 7 articles from the classification test and 5 articles from the NER test set were found in the validation set used for mid-project evaluation. We considered retroactively removing/replacing these 7 articles to address the potentially deleterious effects of such data leakage but decided against these actions for several reasons. First, these articles made up only a very small portion of the mid-project evaluation set, and we did not look at them specifically, so their effects on decisions regarding ML model training/selection were negligible. Second, the only decision made regarding ML training/selection due to the mid-project evaluation was to use precision rather than F1-score for article classification model selection. Due to the very low portion of test set articles in the evaluation (≤1.5%), we are confident that such a decision would have been reached in their absence. Since we believe no portion of the model training or selection process was biased by the presence of these articles, the test sets still serve as representative samples for model evaluation. Third, retroactively removing/replacing these 7 articles would result in convoluting an already complex pipeline and reporting of methods. Ultimately, since our goal was a practical application of ML methods, we opted to simply openly disclose the issue.

3.3.2. Selective manual evaluation of preliminary inventory.

With the results above, the pipeline was implemented and the preliminary inventory up to the point of manual evaluation was created. Of the 3566 records remaining after automated deduplication, 1033 (29.0%) were flagged for review due to a probability score < 0.978, 469 (13.2%) due to potential duplicate names, and 215 (6.0%) due to potential duplicate URLs.

Of the 1033 low probability flags, a curator determined that 805 be retained in the inventory and 228 (6.4% of the total and 22.1% of the flagged) be removed. The majority of those removed, 161/228 (70.6%), were removed because of a partially correct name, while 36/228 (15.8%) were removed for an incorrect name, 27/228 (11.8%) for incorrect classification, and 4/228 (1.8%) for erroneous URLs. As expected, the average probability of those removed (0.749) was much lower than the cut-off that triggered the review.

Of the 469 flagged for duplicate names, 355 (75.7%) were marked for merger, 54 (11.5%) were associated with records that would be removed due to low probability, 50 (10.7%) were not to be merged, and 10 (2.1%) required a partial merger (e.g. sets of > 2 potential duplicates where only some should be merged). Of the 215 flagged for duplicate URLs, 121 (56.3%) were marked for merger, 66 (30.7%) were associated with records that would be removed due to low probability, and 28 (13.0%) were not to be merged.

3.5.1. Countries represented.

Resource locations were assessed by geolocating the URL host IP address and by mining the author affiliations. For 1672 (53.7%) resources at least one IP address could be geolocated, with a total of 1679 IP address locations (Fig 7A). While our methods for identifying countries yielded a consistent output, it misses certain countries, especially for author affiliations where locations are reported more idiosyncratically. For instance, the ISO-3166-1 name for South Korea is “The Republic of Korea”, and if the name does not appear as such in the affiliations, it is missed. Additionally, false positives may occur, such as an affiliation with New Mexico (a state in the USA) being counted as Mexico (the country). However, as a preliminary assessment, 65 unique countries were found in author affiliations (S4 Fig) and 28 unique countries were found in the host IP address locations (S3 Fig). Despite the challenges with cleanly identifying countries, over twice as many countries were identified via author affiliations, indicating that global contributions go well beyond the discrete physical location of a resource. This further highlights the distributed nature of the overall infrastructure.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Biodata resource geolocation metadata.

(A) URL host IP address coordinates (n = 1679). Transparency is constant for all points; darker regions indicate overlapping points. (B) Number of biodata resources per country after determining the country of origin for the most frequently identified funding agencies. Funding agencies that could not be mapped to a single country are not shown; specifically, 111 resources were funded by international agencies. Figures were created using the R ggplot2 package which obtains map data from Natural Earth [71], which is in the public domain.

https://doi.org/10.1371/journal.pone.0294812.g007

3.5.2. Metadata for follow-up analyses of funder and text mining.

Funding of biodata resources can be obtained from the granting agency metadata provided for individual articles. Of the 3705 articles, one or more funding agencies were retrieved for 1916 articles (52.9%), which covers 1714/3112 (55.1%) of the biodata resources identified. Although a total of 1788 “unique” agency names were retrieved, the names reported are variable in that they are often free text values provided by article authors. Both PubMed Central and Europe PMC, which exchange data, make efforts to standardize funder information for their parent funders; however, smaller funders and funders outside of the US and Europe are more likely to be represented irregularly. We see how this challenge is exacerbated in our work by the reporting of increasingly diverse and granular funders as the number of associated biodata resources decreases to only 1 or 2, where reported funder names may be very specific (e.g. an academic department) yet vague (e.g. ambiguous with respect to which university), related to an individual (e.g. scholarships or fellowships of unclear origin), or attributed to a project which, on investigation, is found to be funded itself by multiple funders. Additionally, the funder names retrieved may be of varying organizational levels (e.g. both NIH as a parent and NIH NIGMS as a child), and the same funder can be cited in a single article for distinct grants. In this first instantiation of the inventory itself we left all agency names as is, but future iterations of the inventory may explore use of funder registries, such as the one being developed by CrossRef, to standardize funder names and map identifiers, where possible [72].

To begin assessing the global distribution of the most prevalent funders found here, the funder names associated with 3 or more biodata resources (200/1788, 11.2%) were evaluated by a curator to identify the country or region of the funding organization and verified by a second curator. As noted in prior work, funders themselves are diverse [73], and in addition to government agencies, academic and philanthropic organizations were also identified here. After deduplication of resources, 28 unique counties were identified, with each supporting anywhere from 3 to 570 biodata resources (Fig 7B).

The articles associated with the biodata resources within the inventory also provide opportunities to glean additional information about the resources themselves through text mining. For each article, Europe PMC provides in-house text mining results as annotations that cover database accessions and resource names of over 60 biodata resources as well as gene/protein names, organisms, diseases, chemicals, gene ontology terms and experimental methods [17]. Additional annotations submitted from the community cover specific interactions, targets, pathways, processes, and other terms [74]. Europe PMC offers a dedicated Annotations API for access to all available terms, and new text mining may also be undertaken on full text available via XML. Thus the vast majority of associated articles have text-mined terms already available for further analyses and the majority are available for additional full text analysis (Fig 8).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Inventory text mining potential.

Text mining potential of (A) individual articles found to describe biodata resources, and (B) the collective articles describing a particular biodata resource.

https://doi.org/10.1371/journal.pone.0294812.g008

3.5.3. Overlap with resources found in re3data.org and FAIRsharing.

The machine learning-enabled approach used here represents an effort to identify resources at scale. We initially expected greater overlap with re3data and FAIRSharing, but only 536/3112 (17.2%) inventory resources were identified in these two registries; similarly, the majority of life science resources within both re3data (975/1189, 80.5%) and FAIRsharing (1161/1640, 70.8%) are not found in our inventory.